-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for Arducam OV64A40 64MPx camera module #84
Conversation
Add a new subpoject wrap file for the libpisp library located at https://github.com/raspberrypi/libpisp The libpisp library is used to configure the Raspberry Pi 5 Frontend and Backend ISP components. Signed-off-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Add the Raspberry Pi 5 PiSP specific compressed Bayer format types 1/2: - V4L2_PIX_FMT_PISP_COMP1_xxx - V4L2_PIX_FMT_PISP_COMP2_xxx Add the Raspberry Pi 5 PiSP Frontend and Backend config formats: - V4L2_META_FMT_RPI_FE_CFG - V4L2_META_FMT_RPI_BE_CFG Add the Raspberry Pi 5 PiSP Frontend statistics format: - V4L2_META_FMT_RPI_FE_STATS Add 16-bit Bayer formats: - MEDIA_BUS_FMT_Sxxx16_1X16 Signed-off-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
We can perform some of the local contrast adjustment using global gains in the LSC table. We can vary the amount of gain according to the measured brightness of that image region. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
This was being re-read in order to determine what LSC gains had been applied. We can just retrieve these numbers from the prevAsyncResults_ instead. This will also enable other future algorithms to manipulate the LSC tables in the alsc.status, without it breaking the core ALSC algorithm here. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
"Fast desaturation" is a technique that can help the AGC algorithm to desaturate images more quickly when they are very over-exposed. However, it uses digital gain to do this which can confuse our HDR techniques. Therefore make it optional. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Since noise control handling differs between the VC4 and PiSP IPAs, move the current denoise control handler from ipa base into the vc4 IPA derived class. Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
AWB writes this out during prepare, so we may as well read it in AGC prepare as well. Reading it in process is wrong on the PiSP platform because process runs before prepare, so the AWB status won't be there (on vc4 it made no difference). Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
We need to be able to do things like enable/disable AGC for all the channels, so most of the AGC controls are updated to be applied to all channels. There are a couple of exceptions, such as setting explicit shutter/gain values, which apply only to channel 0. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
We add an HdrMode control (to enable and disable HDR processing) and an HdrChannel, which indicates what kind of HDR frame (short, long or medium) has just arrived. Currently the HdrMode supports the following values: * Off - no HDR processing at all. * MultiExposureUnmerged - frames at multiple different exposures are produced, but not merged together. They are returned "as is". * MultiExposure - frames at multiple different exposures are merged to create HDR images. * SingleExposure - multiple frames all at the same exposure are merged to create HDR images. * Night - multiple frames will be combined to create "night mode" images. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Add support for the following HDR modes in the Raspberry Pi IPA: - Night mode - Single exposure mode - Multi-exposure (merged and unmerged) The algorithm is updated to expect the HDR short channel to meter explicitly for highlights. This means that it will not in general under-expose the short channel more than is actually necessary. The impact on the HDR algorithm is then that this determines how we build tonemaps dynamically. The highlights are more-or-less correct now, so we have to build a power-type curve that gives us the appropriately configured targets in the lower part of the histogram. We allow the tuning file to supply the maximum spatial gain value, rather than the whole curve (though it can supply this if it wants). Some parameter defaults are tweaked to be generally better across the range of our cameras. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> Signed-off-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Add the Raspberry Pi 5 ISP (PiSP) pipeline handler to libcamera. To include this pipeline handler in the build, set the following meson option: meson configure -Dpipelines=rpi/pisp Signed-off-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Add the Raspberry Pi 5 ISP (PiSP) IPA to libcamera. To include this IPA in the build, set the following meson option: meson configure -Dipas=rpi/pisp Signed-off-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
The old ctt.py and alsc_only.py scripts are removed. Instead of ctt.py use ctt_vc4.py or ctt_pisp.py, depending on your target platform. Instead of alsc_only.py use alsc_vc4.py or alsc_pisp.py, again according to your platform. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Added the ability to tune the chromatic aberration correction within the ctt. There are options for cac_only or to tune as part of a larger tuning process. CTT will now recognise any files that begin with "cac" as being chromatic aberration tuning files. Signed-off-by: Ben Benson <ben.benson@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
Changed how users select which platform to tune for. Now users specify a command line argument, '-t', to specify which target platform. Signed-off-by: Ben Benson <ben.benson@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
The various boilerplate parts of the tuning file are extended to include the necessary extra bits for HDR, specifically: * rpi.denoise has different configurations for HDR modes * rpi.agc now has extra channels for HDR * rpi.hdr parameters are added. Signed-off-by: David Plowman <david.plowman@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com>
The IMX708 sensor driver advertises its module variants (narrow/wide angle lens, IR block/pass) by modifying the media entity name string. So add duplicate entries for each variant. Signed-off-by: Nick Hollinghurst <nick.hollinghurst@raspberrypi.com> Signed-off-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: Naushir Patuck <naush@raspberrypi.com> Reviewed-by: David Plowman <david.plowman@raspberrypi.com>
Look for the RAW mandatory stream flag in the pipeline handler config file. If this flag is set, it guarantees that the application will provide buffers for Unicam Image, so override the minUnicamBuffers and minTotalUnicamBuffers config parameters in the following way: - If startup drop frames are required, allocate at least 1 internal buffer. - If no startup drop frames are required, do not allocate any internal buffers. Look for the Output 0 mandatory stream flag in in the pipeline handler config file. If this flag is set, it guarantees that the application will provide buffers for the ISP, do not allocate any internal buffers for the device. Add a new rpi_apps.yaml pipeline handler config file that enables both these flags. To use the file, set the following env variable for a custom build: export LIBCAMERA_RPI_CONFIG_FILE=/usr/local/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml or for a packaged install: export LIBCAMERA_RPI_CONFIG_FILE=/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml Signed-off-by: Naushir Patuck <naush@raspberrypi.com>
Add an entry for the Omnivision OV64A40 Sensor which has a square pixel size of 1.008µm. Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks mostly good here apart from a couple of minor things. I've not run it to look at images or anything, but the numbers seem reasonable.
What's the plan to integrate this? Do you want to send it to the mailing list and then we can tag the commits there?
* OV64a40 doesn't output metadata, so we have to use the "unicam parser" which | ||
* works by counting frames. | ||
*/ | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment seems to be pulled from another helper, but really it's out of date. Probably best to remove it entirely.
double CamHelperOv64a40::getModeSensitivity(const CameraMode &mode) const | ||
{ | ||
/* In binned modes, sensitivity increases by a factor of 2 */ | ||
if (mode.binX >= 2 && mode.scaleX >= 4) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comment doesn't seem to match what the routine is doing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll drop both comments ;)
Yeah I plan to send it to the ml indeed, but only the vc4 part, that's why I went for a pull request here first. What would your preference be ? |
Support the OV64A40 sensor with a camera helper to manage the gain model, light sensitivity, and control delays. Signed-off-by: Lee Jackson <lee.jackson@arducam.com> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
Provide the OV64A40 tuning files for the Arducam Omnivision camera module to operate on the VC4 ISP architecture on Raspberry Pi 4 and below. Signed-off-by: Lee Jackson <lee.jackson@arducam.com> Signed-off-by: Kieran Bingham <kieran.bingham@ideasonboard.com>
Provide the OV64A40 tuning files for the Arducam Omnivision camera module to operate on the PiSP ISP architecture on Raspberry Pi5. Signed-off-by: Lee Jackson <lee.jackson@arducam.com> Signed-off-by: Jacopo Mondi <jacopo.mondi@ideasonboard.com>
24fc53c
to
5fc413f
Compare
I would suggest remove the pisp file, send to the ML to get merged, then once it's merged we rebase on here and include the pisp tuning file. |
Will do! |
"bayes": 1, | ||
"ct_curve": | ||
[ | ||
2300.0, 1.0522, 0.4091, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
2300 is an admirably low value, might your incandescent mode want to stretch down this far? (Currently it stops at 2500.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At present, the lowest level of our light source is 2300k and the next level is 2700k. We expect it to be able to interpolate to a 2500k light source. If 2300k is deleted, it will start from 2700k. Will there be any negative impact from this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No - just that you have no way to get a colour temperature lower than 2500. But if you're happy for the AWB searches to stop at 2500, then that's fine and we are happy to merge this as it is!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No - just that you have no way to get a colour temperature lower than 2500. But if you're happy for the AWB searches to stop at 2500, then that's fine and we are happy to merge this as it is!
Got it, that's acceptable to me.
3000.0, 0.7597, 0.4421, | ||
4000.0, 0.5972, 0.5404, | ||
4150.0, 0.5598, 0.5779, | ||
6500.0, 0.4388, 0.7582 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This isn't a particularly high value, so I wonder if you might want to delete the "cloudy" AWB mode... I can't think it would do anything useful?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, currently our light source can only go up to 6500k, the color temperature range for cloud mode is 7000k to 8000k, I think we can indeed remove that, do you have any suggestions for the calibration process? Especially when there are not many light sources.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it's probably best to remove it. On the other hand, if you've tried it and it actually does something useful, then it's fine to keep it. Getting hold of very blue lamps is tricky, I suppose you could buy some filters but these are often not great either. But I think leaving the top at 6500 is perfectly fine too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it's probably best to remove it. On the other hand, if you've tried it and it actually does something useful, then it's fine to keep it. Getting hold of very blue lamps is tricky, I suppose you could buy some filters but these are often not great either. But I think leaving the top at 6500 is perfectly fine too.
Got it, thanks for your suggestion.
I have sent the VC4 part to the mailing removing the "cloudy" AWB mode. I will send a separate pull request with support for pisp only. |
The OV64A40 is merged upstream in libcamera 0.2.0, but Raspberry Pi have not yet had the opportunity to make a release based on top of that yet. The Pi5 support is expected to be posted to the libcamera development tree very soon, now that the drivers are published, but we're a few weeks away yet from getting that merged still. So unfortunately at present we have the raspberrypi/libcamera tree here with Pi5 support, and the libcamera mainline tree with OV64A40 support, but not a single tree with both. As you have purchased this camera from Arducam, you can find an installation method to support this camera documented on their website. I believe this is where they document it: Once Raspberry Pi make a new release, this step would not be needed. |
Thanks Kieran for the reply. Great work you do on libcamera! So just confirming a few points above:
Thanks for the link to the Arducam docs. I was sold on the idea and advertising that they 'had upstream support' which is plastered all over their website for the OwlSight (OV) cameras, which they don't have for their Sony sensor version (Hawkeye). Would be good if the Sony 64 MP sensor got added as well, but much less likely given Arducam's ability to support their software. All good. I will keep an eye out on when the 0.2.0 update comes through on RPi end ( |
Since Pi 4 support for OV64A40 has been merged upstream, this PR needs a small rework to only add Pi 5 support, plus targeted to the next branch. We are holding off on an official libcamera release until we have Pi5 support upstream, hopefully this should be very soon now. Otherwise it's quite a large maintenance burden working with releases from a downtream tree that is diverged from upstream.
This raises a good point, perhaps @ArduCAM can elaborate a bit. Pi 5 will have upstream/downstream support for OV64A40, but what about the other Arducam sensors? Is there a plan to add IMX519/Hawkeye? It would be quite unfortunate if their users had to use unofficial versions of libcamera for some of the sensors they sell... |
I don't believe the Hawkeye (Sony) driver has ever been sent to the linux-media mailing list, therefore doesn't meet the requirements for being merged into libcamera upstream. Umang has sent the IMX519 driver to the list a couple of times, and hence the tuning and cam_helper have been merged to libcamera. However it has never been updated for AF support or similar. Pivariety modules are a whole issue in themselves. Whilst the Pi kernel does have the driver, support is minimal from our side. |
It would be nice if @ArduCAM can upstream any Pi5 tuning file they have, like for the IMX519. Without it, upstream libcamera released will not run for the sensor. |
This is easy enough for me to test. I'll have a play with the RPi 4 tonight and see if it works out of the box without too much messing around. I did note that the RPi Imager now asks what model Pi you are using. Does this mean I need different SD Card images for old and new Pis? (I guess I'll find out).
I tested the v1.3 camera with the RPi 5 and it worked ok out of the box. Will be good to get all the latest changes for other cameras as well.
As @6by9 said, the Sony Sensor was never met the requirements, so I kind of accepted this wouldn't work with the default stack. It would be good though. I don't think they have stopped selling it, you can still get them on their website. |
So it doesn't run on RPi 4 either. Same problem with OV64A40 and RPi4 as RPi5. I guess I'll be going down the custom / script method with Arducam for now. |
To get this working on a Pi 4 you will need to rebuild libcamera from git.libcamera.org until we have rebased this tree and created release packages. |
Hi @naushir, That's what I ended up doing. I have the camera working. Just making it clear that it is NOT upstreamed in RPi4 either. |
Hey @naushir - just wondering if there are any updates on progressing this? It's been over 2 weeks since we last messaged. I am in conversation with Dion at @ArduCAM about this PR as well to see if more support and resources can be thrown at this PR or merging the git.libcamera.org into Raspberry Pi repos going forward. |
This has now been merged in our downstream release of libcamera. You can get this with an apt update. I'm closing this issue down now. |
No description provided.